Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer

We address the challenging problem of Natural Language Comprehension beyond plain-text documents by introducing TILT neural network architecture which simultaneously learns layout information, visual features, and textual semantics. Contrary to previous approaches, we rely on a decoder capable unifying variety problems involving natural language. The is represented as an attention bias complemented with contextualized while core our model pretrained encoder-decoder Transformer. Our novel approach achieves state-of-the-art results in extracting information from answering questions demand understanding (DocVQA, CORD, SROIE). At same time, simplify process employing end-to-end model.